btc/dgb/bch: gate share broadcast on tx completeness + mark only what was sent - #880
Conversation
593401f to
cd12988
Compare
cd12988 to
974c3dc
Compare
|
🛑 BLOCKING — do not merge. This PR wires its core feature into code that can never execute. I was about to merge this as a green non-draft. It is not safe. The defect#880 adds three new instances of the if constexpr (requires { obj->m_new_transaction_hashes; })Added in: That expression is ALWAYS FALSE in all three lanes. Verified on master:
The member is nested at Consequence: It is also a REGRESSION on btc
There is even a comment at Why CI cannot catch thisThis is the A passing test suite is not evidence here. Please state explicitly whether Required before this can merge
What is good hereThe gate semantics are right, and the header comments are unusually clear — particularly "gate on the BYTES in Re-request review once the probe is fixed. This is otherwise close. |
NodeImpl::send_shares gated its remember_tx/forget_tx relay on a top-level obj->m_new_transaction_hashes that no dgb share type declares -- the field is nested in m_tx_info (dgb::ShareTxInfo). The probe was therefore always false, so DGB never forwarded a referenced new-tx hash to a peer for any share version (v17..v36). Route the collection through a new SSOT, dgb::append_share_tx_refs, which guards on m_tx_info: live for v17/v33 (which carry it), compiled out for v34/v35/v36 (which do not) -- correct-by-construction, never silently dead. Faithful port of the btc fix behind #880. Adds a fails-before KAT (dgb_share_tx_relay_refs_test) registered in the CI --target allowlist: reverting the probe to the top-level member collapses the v17/v33 expectations to zero. Co-authored-by: frstrtr <frstrtr@users.noreply.github.com>
… was sent Two reward-critical share-loss fixes, ported to BTC, DGB and BCH from the already-correct DASH implementation (src/impl/dash/node.cpp). F3 -- tx-completeness broadcast gate. send_shares() collected the referenced new-tx bytes with `if (it != m_known_txs.end()) full_txs.emplace_back(...)` and no else branch: a tx we do not hold was silently omitted and the share was written to the peer anyway. Canonical p2pool then drops the connection on "referenced unknown transaction" (p2p.py), which isolates us from the sharechain and orphans our shares. Shares pulled via sharereply arrive without their tx bytes, so this is reachable in normal operation. send_shares now filters the outgoing batch through partition_backable() (gate on the BYTES in m_known_txs, not on the peer's possibly-stale have_tx advert) and the now-unreachable omission path increments a counter that logs a warning. F2 -- mark only what was sent. broadcast_share() inserted each walked hash into m_shared_share_hashes INSIDE the chain-walk loop, before send_shares ran. send_shares returned void and has abandon paths (tracker try_to_lock miss, empty batch, and now the F3 gate), and zero connected peers abandons the batch too. A marked-but-unsent share is lost permanently: the next walk breaks on the first marked hash, so nothing ever re-pushes it -- silent PPLNS-credit loss with no retry path. send_shares now returns the hashes it actually wrote and broadcast_share marks only those, after the peer loop, via broadcast_and_mark(). m_last_broadcast_to likewise records what was written rather than what was offered, so a withheld share is never blamed for a peer disconnect (which would mark it rejected and block all future re-broadcast). Consensus surface: NONE. Both changes decide only WHETHER a share goes on the wire. Share bytes, minting, PPLNS weighting and payout are untouched, and a withheld share stays in the chain and is re-offered once its txs arrive. Primitives follow the per-coin header pattern established for BTC in #868: src/impl/{btc,dgb,bch}/known_txs_retention.hpp. The LTC lane is concurrently hoisting all_txs_backable into src/core/; these four per-coin headers (including dash) should be collapsed onto that hoist once both land. Tests ride existing allowlisted CI targets -- no new add_executable, no build.yml change: btc -> btc_share_test (broadcast_gate_test.cpp) dgb -> dgb_share_test (broadcast_gate_test.cpp, known_txs_retention_test.cpp) bch -> bch_embedded_block_broadcast_test (broadcast_gate_test.cpp; the bch test tree has no GTest harness, so the TU exposes a checks function the host main() calls) Mutation-verified red: deleting the gate reddens 2 btc / 4 bch assertions; restoring mark-before-send reddens 4 btc / 5 bch assertions.
… pre-wiring A caller-side reachability trace shows DGB is not symmetric with btc/bch: main_dgb.cpp binds no create/mint-share fn, so dgb::NodeImpl::broadcast_share and notify_local_share have zero callers repo-wide -- DGB never mints a local share today (#884). DGB's live path into send_shares is readvertise_best_share() (ROOT-2 re-advert on best-change plus a 10s one-shot timer), which re-pushes PEER-RECEIVED shares and deliberately bypasses the de-dup set. Verified independently: set_mint_share_fn / set_create_share_fn / broadcast_share / notify_local_share have zero occurrences in main_dgb.cpp; the only set_mint_share_fn binding in the tree is main_dash.cpp:1854 plus dgb's own work_source_test. readvertise_best_share is called from node.cpp:1767 and 1774. BTC (main_btc.cpp:1336 after lk.unlock() at :1323) and BCH (pool_entrypoint.hpp:564 after lk.unlock() at :562) are both live and unchanged. Changes: - dgb: extract the readvertise recording rule into readvertise_and_record() and wire readvertise_best_share() through it. This is DGB's live reward-critical marking discipline: m_last_broadcast_to is what a peer disconnect within 10s converts into m_rejected_share_hashes, and the readvertise walk `continue`s past rejected hashes permanently. Recording the OFFERED batch would let a share the F3 gate merely withheld be blamed for someone else's drop and excluded from every future re-advert -- a propagation loss for the whole sharechain, since DGB re-advertises peer-received shares. - dgb: rescope the KATs onto that live path. The gate cases (partition_backable) and the new DgbReadvertiseRecording cases drive code production executes; the broadcast_and_mark cases are renamed DgbBroadcastMarkingPrewired_NotReachedToday so no reader mistakes them for live protection. Removes the broadcast_share de-dup-walk KAT, which modelled an unreachable function -- the shape of the LTC #873 miss. - dgb: label NodeImpl::broadcast_share in source as pre-wiring for #884. The F2 fix stays so the seam is correct when the mint path is bound and so the three coins remain symmetric, but it protects nothing at runtime today. - btc/bch: document the ROOT-2 interaction at readvertise_best(). Marking is now a strict subset of what the pre-fix code marked, so the walk breaks later or in the same place, never earlier: the re-advert re-pushes a superset of what it used to. Monotonically better, strictly better for head shares minted while no peer was connected. It does not fully close ROOT-2 here -- shares already accepted by another peer stay marked and the walk still breaks, as before. That needs the ltc/dgb-style de-dup-bypass readvertise; pre-existing, out of scope. No behaviour change in this commit for btc/bch. Mutation-verified red on the DGB live path: recording the offered batch instead of what was written reddens WithheldShareIsNeverBlamedForAPeerDrop; dropping the empty-send skip reddens PeerThatReceivedNothingIsNotRecorded; deleting the gate reddens 2 DgbBroadcastGate cases.
…coin new_tx_hashes SSOT The old flat m_new_transaction_hashes probe was FALSE for every share variant (v17/v33 nest the refs in m_tx_info; v34+ carry none), so the F3 gate collected empty refs and treated every share as vacuously backable -- it never withheld an incomplete share on any version. Route each coin through its SSOT accessor from #913/#914 (btc/bch::new_tx_hashes, dgb::append_share_tx_refs) so the gate sees the real refs. Adds BtcBroadcastGate.RealShareTxInfoRefsWithheld driving a REAL v17/v33 share (not a top-level mock); mutation-verify confirms it goes red when the probe is reverted to the dead flat form.
974c3dc to
d3e5151
Compare
…bility Instrument btc::NodeImpl::broadcast_share with the deferred/acquired/ reached-send counters and IsNull/contains guards that ltc already carries, and add broadcast_lock_discipline_test.cpp driving a REAL btc::NodeImpl against a populated sharechain. Proves the F3 send path (tx-completeness gate + send_shares + mark-after-send) is reachable, not dead code behind an always-held lock: reached_send is 0 when the caller holds the tracker mutex EXCLUSIVELY on the calling thread and 1 when it does not -- the shape main_btc create_share_fn guarantees by dropping the exclusive lock (lk.unlock) before calling broadcast_share. Rides the allowlisted btc_share_test target (no new add_executable, #769).
|
Still blocked — partially fixed, and the partial state is worse than either extreme. Re-measured on the current head: One of the four sites was corrected; three still carry the always-false probe. That is a worse outcome than the original, because the PR now looks fixed on a spot-check while two thirds of the F3 gate remains inert. Both reference patterns are on master — copy one
Why this cannot merge green
Please also state whether Required to unblock
The gate semantics and the header comments in this PR are good — particularly "gate on the BYTES in |
|
Converted to DRAFT to prevent accidental merge. Not a judgement on the work — a guard. I cannot use Unchanged across ~20 hours and several pushes; the latest commit improves lock-discipline tracing while the core defect stands. One of four sites is correct — a partial fix is worse than none, because it survives a spot-check while two thirds of the F3 gate stays inert. 41 passing checks confirm the code compiles, not that it runs. Mark it ready again the moment all four sites use the correct form — both patterns are already on master (#914 Also still needed: a test that fails if the probe goes false, and confirmation that The gate semantics and header comments in this PR are good and I want it to land. Only the member access is wrong. |
|
Reviewed. My framing of this PR was wrong, and the "3 of 4 probes still dead" measurement was a grep artifact. Correcting the record. All four probe sites already carry the correct nested-first form as of
The grep was counting prose and the intended fallback branch as defects. Live probe sites verified reachable, with caller-side lock traces on all three lanes — no exclusive lock is held at any call site (btc What WAS genuinely missingItems 2 and 3 of the unblock list were done for BTC only. BCH and DGB gate tests drove This was proven by mutation, not asserted. Reverting all three accessors to the flat probe: Every pre-existing Branch:
|
What
Wires the F3 tx-completeness broadcast gate into the actual send path and fixes the F2 mark-before-send share loss, for BTC, DGB and BCH. Both fixes are ports of the already-correct DASH implementation (
src/impl/dash/node.cpp). Builds on #868 (c77dc03), which landed the BTC primitives header.FIX A — F3 tx-completeness broadcast gate
send_shares()built itsremember_txforward withwith no else branch: a referenced tx whose bytes we do not hold was silently omitted and the share was written to the peer regardless. Canonical p2pool then drops the connection on
referenced unknown transaction(p2p.py), which isolates us from the sharechain and orphans our shares.This is reachable in normal operation: shares pulled via
sharereplyarrive without their tx bytes.send_shares()now filters its outgoing batch throughpartition_backable()before buildingneeded_txs. The gate keys on the bytes inm_known_txs, not on the peer'shave_txadvert — that advert can be stale (the peer may have dropped the tx), and sending hash-only for a tx the peer no longer holds is exactly the disconnect.m_remote_txsis still used only to choose the hash-vs-bytes encoding. The now-unreachable omission branch increments amissingcounter and logs a warning (defence in depth, mirroring DASH).Live on all three coins.
FIX B — mark / record only what was actually sent
send_shares()returnedvoid, so callers could not tell what it wrote. It now returns the hashes actually written. Two callers consume that:broadcast_share()(BTC, BCH live; DGB pre-wiring) inserted each walked hash intom_shared_share_hashesinside the chain-walk loop, beforesend_sharesran.send_shareshas abandon paths — trackertry_to_lockmiss, empty batch, and now the F3 gate — and a zero-peerm_peersabandons the batch too. A marked-but-unsent share is lost permanently: the next walk breaks on the first marked hash, so nothing ever re-pushes it. It now marks only accepted hashes, after the peer loop, viabroadcast_and_mark().readvertise_best_share()(DGB only — DGB's live path) recorded the offered batch intom_last_broadcast_to. That record is what a peer disconnect within 10 s converts intom_rejected_share_hashes, and the readvertise walkcontinues past rejected hashes permanently. Recording the offered batch lets a share the F3 gate merely withheld be blamed for someone else's disconnect, turning a transient, self-healing skip into permanent exclusion from every future re-advert. Since DGB re-advertises peer-received shares, that is a propagation loss for the whole sharechain, not just for us. Now extracted intoreadvertise_and_record(): records what was written, and records nothing at all for a peer that received nothing. BTC/BCHm_last_broadcast_togets the same treatment insidebroadcast_and_mark.Reachability (caller-side trace — verified independently)
broadcast_sharereached?send_sharesreached viamain_btc.cpp:1336, afterlk.unlock()at:1323broadcast_share,readvertise_bestpool_entrypoint.hpp:564, afterlk.unlock()at:562(not inmain_bch.cpp)broadcast_share,readvertise_bestmain_dgb.cppbinds no create/mint-share fnreadvertise_best_shareonlybroadcast_sharehalf is pre-wiringDGB detail, all re-verified against this branch:
set_mint_share_fn,set_create_share_fn,broadcast_share,notify_local_share— zero occurrences inmain_dgb.cpp. The onlyset_mint_share_fnbinding in the tree ismain_dash.cpp:1854, plus DGB's ownwork_source_test.cpp.dgb::NodeImpl::broadcast_shareandnotify_local_sharehave zero callers repo-wide (definitions and comments only).try_mint_sharealways takes the null-fn branch atdgb/stratum/work_source.cpp:810-814and logsset_mint_share_fn unbound. Tracked as dgb: set_mint_share_fn is never bound in main_dgb.cpp — DGB cannot mint local shares #884.readvertise_best_share()is live: called atdgb/node.cpp:1767on best-change and armed on a 10 s one-shot timer at:1774, both from the think IO-phase.What that means for DGB in this PR
Chosen: keep the DGB
broadcast_sharehalf, labelled as pre-wiring. Rationale: dropping it would leave a known mark-before-send bug in a function that #884 is going to bind, so whoever wires the mint seam would ship the loss; and the repo's per-coin symmetry posture means the threenode.cppbodies should stay identical. It is zero-risk because it is dead. It is labelled in three places so nobody mistakes it for protection:PRE-WIRING — NOT REACHED ON DGB TODAY (#884)block abovedgb::NodeImpl::broadcast_share;DGB REACHABILITYblock insrc/impl/dgb/known_txs_retention.hppmarking each primitive LIVE vs PRE-WIRED;DgbBroadcastMarkingPrewired_NotReachedToday.DGB tests were rescoped onto the reachable path. The previous revision's DGB KAT modelled the
broadcast_sharede-dup walk — i.e. an unreachable function. That is the shape of the #873 LTC miss, where the test and its negative control both passed against code production never runs. It is replaced byDgbReadvertiseRecording, which drivesreadvertise_and_record— the primitivereadvertise_best_shareactually calls. No DGB case constructs or broadcasts a locally minted share.BTC/BCH ROOT-2 re-advert under the new marking rule
Asked for explicitly, and checked.
btc/node.cpp:1053-1061andbch/node.cpp:1056-1064implementreadvertise_best()as a plain call intobroadcast_share, so the ROOT-2 walk still breaks on the first hash inm_shared_share_hashes— unlike LTC/DGB, which have a dedicated de-dup-bypassingreadvertise_best_share().The change is monotonically safe. Post-fix marking is a strict subset of pre-fix marking (only hashes a peer actually accepted). The walk therefore breaks later or in the same place, never earlier, so ROOT-2 re-pushes a superset of what it used to. Specifically:
So BTC/BCH ROOT-2 behaves as intended and is improved, but this PR does not fully close it — that still needs the LTC/DGB-style de-dup-bypass readvertise. Pre-existing gap, out of scope, documented in a comment at both call sites. No behavioural change was made to
readvertise_best()itself.Consensus / reward safety
No consensus or minting semantics change. Every change decides only whether we put a share on the wire. Share bytes, minting, PPLNS weighting and payout are untouched. A withheld share stays in our chain and is re-offered on the next broadcast/re-advert once its tx bytes arrive.
Skipping costs no propagation: a share we downloaded is by definition already on the network we fetched it from.
Verified file:line claims (corrections to the issue text)
746-750748-750(theelse {opens at 746)726-730728-730749-753751-753broadcast_share@792, mark @823broadcast_share@772, mark @803broadcast_share@795, mark @826Additional findings not in the issue:
send_shares/broadcast_sharebodies were byte-identical across btc/dgb/bch, so one patch applied cleanly to all three.send_sharestakes the trackershared_lockitself (DASH's caller holds it). Left as-is.send_sharescall site —readvertise_best_share()— that BTC/BCH lack. It turns out to be DGB's only live one.register_template_txs/ rolling template window and noadd_local_share; they usecore::evict_known_txs_to_cap+m_known_txs_order. Soretain_template_txsships (for btc: F3 known-txs retention/backable primitives + unit KAT #868 symmetry, with KAT coverage) but is not wired, and there is no decline-mint call site to port — the DASHadd_local_sharedecline has no analogue here.Tests
Folded into existing allowlisted CI targets — no new
add_executable, no.github/workflowschange (check_test_target_allowlist.pypasses):btc_share_testbroadcast_gate_test.cppdgb_share_testbroadcast_gate_test.cpp,known_txs_retention_test.cppbch_embedded_block_broadcast_testbroadcast_gate_test.cppThe bch test tree is plain
int main()/assert with no GTest, so a second source cannot carry its ownmain(). The BCH TU instead exposesrun_share_broadcast_gate_checks()and the hostmain()calls it — same "ride an allowlisted executable" pattern, no NOT_BUILT sentinel.Mutation evidence (tests are load-bearing)
Gate deleted (
if (all_txs_backable(...))->if (true)):Pre-fix F2 restored — mark the offered batch instead of what was sent (btc/bch, the live
broadcast_sharepath):DGB live path (
readvertise_and_record) — record the offered batch instead of what was written:DGB live path — drop the "peer that received nothing gets no record" skip:
broadcast_and_mark()andreadvertise_and_record()both deliberately take the offered batch as a parameter, so "record/mark the offered batch" is expressible at the primitive level and therefore detectable by the KAT.Green runs (local, gcc-13 Release, same conan profile as CI)
Binaries build clean:
c2pool(-DCOIN_DGB=ON) andc2pool-bch(-DCOIN_BCH=ON), so all threenode.cppTUs compile.Not verified
broadcast_share/readvertise_best_sharehaving no other write intom_shared_share_hashes/m_last_broadcast_to.sharereply-pulled shares.For the integrator — follow-up de-duplication
A sibling lane is hoisting
all_txs_backableintosrc/core/for LTC. This PR deliberately does not create a competingsrc/core/copy; it follows #868's per-coin header pattern. Once both land there will be four per-coin copies (btc,dgb,bch,dash) plus the core hoist — these should be collapsed onto thesrc/core/version in a follow-up. A note to that effect is in each header.readvertise_and_record()is intentionally DGB-only, since DGB is the only coin with a de-dup-bypassing re-advert.Ownership fences respected
Nothing touched in
src/core/socket.cpp,src/core/web_server.cpp,src/impl/ltc/,src/impl/dash/, or anyprotocol_legacy.cpp.